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1.0  INTRODUCTION 


An  explanation  of  the  title  of  this  effort,  “Complementary  Advanced  Fusion  Exploration 
(CAFE)”  is  in  order  at  the  outset.  “Complementary”  refers  to  the  notion  of 
supplementing  and  aiding  the  research  in  efforts  directly  related  to  improving  the  state  of 
mathematical  applications  to  fusion.  These  efforts  are  limited  to  those  in  the  Fusion 
Technology  Branch,  and  more  specifically,  those  being  monitored  by  the  author.  Some 
reference  to  a  broader  class  of  fusion  problems  may  also  be  addressed  in  the 
development.  “Advanced”  refers  to  sets  of  problems  and  solutions  not  typically 
considered  by  fusion  researchers.  That  is,  The  goal  here  is  to  “fill  in  some  blanks”  with 
respect  to  new  ideas  in  sensor,  data,  information,  and  other  forms  of  fusion. 

“Exploration”  is  not  to  be  confused  with  “Exploitation.”  This  term  refers  to  the  notion  of 
examining  the  mathematical,  physical,  and  engineering  literature,  attending  talks  by 
foremost  researchers,  and  keeping  abreast  of  the  most  recent  developments  of  those 
researchers.  This  has  been  the  attempt  of  this  author  during  this  effort  together  with  “out 
of  the  box”  thinking. 

The  beginning  of  this  effort  dealt  with  examining  specific  archives  of  information, 
including  the  streaming  video  available  from  the  Mathematical  Sciences  Research 
Institute  (MSRI).  Additionally  http://www.kitp.ucsb.edu/  online  talks  were  of  great 
use,  as  was  the  MIT  Open  Courseware  and  other  sources.  These  videos  are  considered 
extremely  valuable  and  the  reader  is  encouraged  to  explore  them  It  also  included  a  look  at 
categories  within  arxiv.org  and  citeseer.  At  the  end,  the  author  was  led  back  to  PhD 
Theses,  IEEE  articles,  SPIE  articles,  IEE  articles,  and  Books  available  from  the  AFRL 
Technical  Library. 

As  part  of  the  effort,  three  summer  students  were  hired  to  assist  the  engineer.  The 
result  of  their  work  led  to  a  look  at  Unmanned  Aerial  Vehicle  Swarm  Path  Planning 
analysis,  Kalman  Filtering,  Trifocal  Tensor  and  Bundle  Adjustment,  and  Game  Theoretic 
research,  and  finally,  particle  filtering. 

This  report  documents  the  work  performed  on  an  in-house  effort  entitled 
“Complementary  Advanced  Fusion  Exploration.”  The  effort  began  with  a  thorough 
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canvassing  of  various  mathematics  and  science  literature  to  detennine  what  new 
techniques  could  be  applied  to  the  area  of  Information  Fusion.  The  result  of  this  research 
led  to  a  focus  in  the  areas  of  particle  filtering,  force  aggregation  using  Probability 
Hypothesis  Methods  (PHM),  and  Data  Mining  of  Ground  Moving  Target  Indicator 
(GMTI)  Databases. 

The  overall  objective  of  the  effort  was  to  complement  existing  and  past  efforts 
that  have  been  applied  to  the  Airborne  Warning  and  Control  System  (AWACS),  the  Joint 
Strike  Fighter  (JSF),  Multimission  Command  and  Control  Systems  (MC2S),  and  Space 
Based  Radar  (SBR).  MC2S  includes  both  the  Multimission  Command  and  Control 
Aircraft  (MC2A)  and  the  Multimission  Command  and  Control  Constellation  (MC2C). 

Among  these  was  an  effort  performed  by  Andro  Consulting  Service  (ACS)  that 
Dr.  Donald  Weiner,  Dr.  Pramod  Varshney,  and  Dr.  Ruixin  Niu,  all  of  Syracuse 
University  at  the  time.  Dr.  Niu  had  written  his  thesis  entitled  Practical  Issues  in  Target 
Tracking  under  the  advisement  of  Dr.  Yaakov  Bar-Shalom  and  Dr.  Peter  Willett,  of  the 
University  of  Connecticut.  The  effort  involved  a  trip  to  the  AWACS  System  Program 
Office  (SPO),  to  brief  them  on  the  idea  of  using  AFRL’s  JVIEW  Tool  for  the  AWACS 
Block  Upgrade  being  considered  at  the  time.  One  of  the  things  that  came  out  of  that 
meeting  was  the  archaity  of  the  AWACS  Computing  Capabilities.  The  idea  was  to 
improve  the  AWACS  picture  by  enabling  the  AWACS  to  visualize  weather  as  detected 
by  onboard  sensing  equipment  and  discern  targets  in  adverse  environments.  The  ideas 
were  all  good  but  the  effort  did  not  proceed  to  Phase  II  due  to  funding  limitations. 

Other  efforts  that  were  capitalized  upon  included  Phase  I  and  Phase  II  SBIRs  that 
were  perfonned  by  Dr.  Daniel  H.  Wagner’s  (DHW),  C.  Allen  Butler,  Dr.  W.  Reynolds 
Monach,  and  Scientific  Systems  Company  Incorporated  (SSCI).  Dr.  Ronald  Mahler  was 
the  consultant  on  the  SSCI  effort.  Dr.  Butler  teaches  a  course  in  Kalman  Filtering.  Dr. 
Monach  was  involved  in  developing  non-Gaussian  tracking  algorithms  that  involved  a 
Monte-Carlo  approach.  Several  other  areas  were  studied  by  DHW.  These  included  a 
voice  interface  for  the  AWACS  operator  using  ViaVoice,  Decision  Aids  for  the  AWACS 
operator,  bearings  only  tracking,  Bayesian  Networks  for  Combat  Identification  (CID), 
and  Genetic  Algorithm  (GA)  AWACS  Flight  Path  Optimization  (FPO).  When  the  idea  of 
GA-FPO  was  brought  in  front  of  the  AWACS  SPO,  they  basically  indicated  that  it  was 
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unlikely  that  the  AWACS  Crew  would  be  receptive  to  altering  their  flight  paths  despite 
the  potential  increase  in  ground  coverage  area. 

SSCI’s  work  dealt  with,  among  other  things,  Unified  Bayesian  Situation 
Assessment  Sensor  Management.  They  actually  implemented  a  particle  filter  in  their 
research.  The  most  recent  Phase  I  effort  has  proceeded  to  Phase  II  and  offers  promise  in 
the  area  of  Force  Aggregation  using  Probability  Hypothesis  Methods. 

In  addition  to  the  contractual  small  business  efforts,  two  summer  students  were 
involved  in  conceptualizing  research  ideas.  Richard  Hartz  did  some  UAV  Swarm  Path 
Planning  research,  and  Melissa  Neumann  (ref.  Appendix  E)  implemented  Kevin 
Murphy’s  Kalman  Filter  and  researched  Game  Theory  Level  III  fusion,  and  Trifocal 
Tensor  Estimation. 

Most  recently,  the  focus  has  been  on  particle  filtering.  There  is  a  need  to  inform 
operational  users  of  the  Kalman  Filter  of  the  benefits  of  particle  filtering. 

The  seminal  paper  on  particle  filtering  is  [2].  Another  useful  paper  on  particle 
filtering  is  [3].  This  paper  points  out  the  increasing  importance  of  including  the  elements 
of  nonlinearity  and  non-Gaussianity.  This  is  especially  true  in  military  applications 
where  the  world/environment  is  definitely  not  linear  or  Gaussian.  In  [3]  seven  particle 
filtering  algorithms  are  described.  These  are  as  follows. 

•  Sequential  Importance  Sampling  (SIS) 

•  Resampling  Algorithm 

•  Generic  Particle  Filter 

•  Sampling  Importance  Resampling  (SIR) 

•  Auxiliary  Particle  Filter 

•  Regularized  Particle  Filter 

•  “Likelihood”  Particle  Filter 

Each  of  these  particle  filters  has  its  own  distinct  advantages  and  disadvantages,  spelled 
out  in  the  paper. 
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2.0  PARTICLE  FILTERING  EQUATIONS 


The  following  equations  [2-4]  are  used  in  particle  filtering  (ref.  appendix  B). 


p(xk  |  Zi:W)  =  J p(xk  |  xk_x)p(xk_x  I  z1:t.j )dxk_l 


(1) 


p(*k  I  Zu) 


/7(zt  |  XjpU,  |  z,,_|) 


(2) 


where  we  have  used  the  notation  z\±  to  denote  the  set  of  measurements  Jz,:  i=l,...,k}. 
This  constitutes  the  available  information  at  time  step  k.  The  first  equation  is  the 
prediction  step.  Particle  filtering  uses  Markov  Chain  Monte-Carlo  (MCMC)  method  to 
draw  a  sample  of  the  probability  density  function  (pdf)  of  the  system  noise.  It  assumes 
that  we  have  a  set  of  random  samples  from  the  pdf  of  under  the  integral  sign  of  ( 1 ), 
namely  p(xk-i|zi:k-i)-  The  particle  filter  propagates  these  samples  according  to  (1)  and 
then  updates  the  samples  according  to  (2)  to  obtain  a  set  of  values  that  are  approximately 
distributed  as  p(xk|zi:k).  Quoting  from  [2],  during  the  prediction  step:  “Each  sample  is 
passed  through  the  system  model  to  obtain  samples  from  the  prior  at  time  step  k.”  The 
system  model  referred  to  is  the  familiar: 

4  (0 = fk- 1  (**_i  (0.  wk~  i  (0)  0) 

“where  Wk-i(i)  is  a  sample  drawn  from  the  pdf  of  the  system  noise  p(Wk-i).”  The  function 
f  is  the  system  transition  function  that  propagates  the  state  vector  x  from  time  k- 1  to  time 
k.  During  the  update  step,  the  measurement  Zk  is  used  to  “obtain  a  nonnalized  weight  for 
each  sample.”  The  nonnalized  weight  is  denoted: 

(4) 

YjP(zk  \xlu)) 

j= i 
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where  q;  is  the  “probability  mass  associated  with  element  i.”  This  original  particle  filter 
was  referred  to  as  the  “bootstrap  filter.”  It  approximates  densities  by  a  finite  weighted 
sum  of  N  Dirac  densities  centered  on  elements  of  Rn  called  particles  [5].  Rn  denotes  real 
n-tuple  space.  That  is,  the  space  of  n-tuples  of  real  numbers  (where  n  is  a  positive 
integer),  ft  is  an  extension  of  the  space  R  .  The  space  R  consists  in  the  set  of  all  real  3- 
tuples  (all  ordered  sequences  of  three  real  numbers)  with  addition,  scalar  multiplication 
and  the  additive  identity  element  (0,  0,  0)  defined  on  the  space  as  well  as  all  the  other 
requirements  for  a  linear  space. 

3.0  GRID-BASED  FILTERS 

Grid-based  filters  use  numerical  integration  for  dynamic  state  estimation.  They 
evaluate  the  pdf  of  the  state  over  a  grid  in  state  space  [2],  The  assumption  is  that  the  state 
space  is  discrete  and  consists  of  a  finite  number  of  states  [3].  The  posterior  pdf, 
p(xk-i|zi:k-i),  is  written  as  a  weighted  sum  of  delta  functions.  This  weighted  sum  is  then 
substituted  into  (1)  and  (2)  to  derive  the  prediction  and  update  equations  used,  and  the 
expression  for  the  weights  comes  from  making  this  substitution.  The  result  is  very 
similar  to  (4). 

4.0  PROBABILITY  HYPOTHESIS  DENSITY  (PHD)  PARTICLE  FILTERING 

The  Swedish  Defense  Research  Agency  is  investigating  the  following  Information 
Fusion  Methods  [7], 

•  Force  Aggregation 

•  Tracking 

•  Sensor  Resource  Management 

As  part  of  that  research  they  are  focusing  on  analyzing  intelligence  reports  at  the  division 
level  in  a  ground  warfare  scenario.  Their  force  aggregation  approach  involves  clustering 
and  classification.  For  ground  vehicle  tracking  they  are  using  a  PHD  Particle  Filter. 

Their  sensor  resource  management  uses  random  set  simulations. 

They  view  the  problem  of  tracking  of  a  large  number  of  vehicles  in  terrain 
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from  incomplete  observations  as  having  the  solution  of  PHD  particle  filtering.  Figure  1 

shows  their  implementation  of  this. 


Figure  1.  Ground  Target  Tracking  using  Airborne  PHD  Particle  Filter 


They  track  the  first  moment  of  joint  distribution,  i.e.,  PHD.  The  integral  of  PHD  over  an 
area  is  the  expected  #  targets.  This  avoids  combinatorial  explosion  and  is  therefore  good 
for  large  number  of  vehicles.  The  particle  filter  implementation  does  not  require 
analytical  motion  nor  observation  models  and  is  suitable  for  non-linear  problems. 

SSC1  has  a  Matlab  implementation  of  a  PHD-PF  that  they  agreed  to  provide  to 
Air  Force  Research  Laboratory’s  Fusion  Technology  Branch.  All  that  is  needed  is 
someone  to  take  the  Matlab  code  and  implement  it  in  the  laboratory  using  real  or  more 
realistic  data.  This  has  the  potential  of  improving  tracker  performance  based  on  the  fact 
that  real  data  is  generally  nonlinear  and  non-Gaussian.  Not  only  that,  but  the  distribution 
of  the  system  and  measurement  noise  is  usually  an  unknown,  and  the  PHD-PF  is  able  to 
handle  system  or  measurement  noise  of  any  distribution. 
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5.0  PARTICLE  FILTERING  AND  REAL  DATA 


The  arena  of  particle  filtering  has  grown  since  its  inception  in  1993.  Acceptance 
Within  the  Air  Force  fusion  community  has  been  somewhat  slow.  There  is  a  plethora  of 
raw  data  to  which  particle  filtering  could  be  applied.  A  good  next  step  would  be  to  gather 
or  acquire  some  real  data  and  try  various  particle  filters  out.  Clearly,  the  real  world  is 
nonlinear  as  well  as  non-Gaussian. 


6.0  PARTICLE  FILTERING:  THE  BASIC  ALGORITHM 

The  basic  algorithm  for  particle  filtering  as  implemented  consists  of  the  following 
assumptions  and  steps.  The  first  assumption  is  that  the  form  of  equation  (3), 
hereafter  referred  to  as  the  “formula”,  along  with  a  distribution  of  the  noise  is 
known.  A  major  challenge  is  to  come  up  with  these  initial  assumptions. 
Kitagawa’s  formula  is  the  basic  one  used  in  the  original  paper  [2],  and  in  the 
tutorial  [3].  There  are  several  other  formulas  available  from  the  economics 
literature.  Another  way  to  model  this  formula  is  to  use  the  classic  linear  state 
transition  function,  but  with  the  nonlinear  control  input  maneuver  models  [22]. 
The  form  of  equation  (3)  used  in  the  Kalman  Filter  is  well  known  to  be 


Where: 


And 


At  =^At-i  +%■!  +>A) 

1  T  0  O' 

0  10  0 
0  0  1  T 

0  0  0  1 


(5) 
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Also  u  and  w  can  be  treated  as  the  acceleration  and  noise  models  of  the  target.  That  is 
u=(ax,  ay). 

Getting  back  to  the  basic  algorithm,  the  predicted  state  is  simply  an 
implementation  of  equation  (3),  with  a  sample  from  the  desired  noise  distribution  added. 
This  is  followed  by  a  calculation  of  the  importance  weights  which  begins  with  a 
calculation  of  the  likelihoods  for  the  particles.  The  likelihood  used  in  [23]  is  as  follows: 


(6) 


w, 


It  was  normalized  by  dividing  each  weight  by  the  sum  over  i  of  the  all  the  weights. 

One  of  three  resampling  techniques  (residual,  systematic,  or  multinomial,  are  then 
applied,  and  the  time  for  the  total  calculation  is  computed.  This  enables  an  evaluation  of 
performance  versus  resources. 

In  the  results  obtained  in-house,  the  Unscented  Kalman  Filter  outperformed  the 
basic  PF  algorithm,  and  the  Unscented  Particle  Filter  (UPF)  did  the  best  (ref.  fig.  2). 


RMS  Error  Comparison  for  Various  Filtering  Methods 


80 


70 


60 


50 
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PF-UKF 


PF-UKF-MC 


Filtering  Method 


Figure  2.  Results  of  in-house  analysis,  showing  superior  performance  of  UPF. 
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APPENDIX  A  -  DENOMINATOR  OF  SECTION  1  EQUATION  2 

The  denominator  of  Equation  (2)  is  comes  from  the  Total  Probability  Theorem 
(TPT)  and  is  given  by 

P(zk  I  zv.k-i)  =  J p(zk  I  xk)p(xk  I  Zhk_x)dxk  (Al) 

Giving 


P(xk  I  zu)  = 


p(zk 

\xk)p{xk 

Z\:k-\) 

\p(zk\ 

*k)p(xk  1 

zv.k-x)dxk 

(A2) 


This  comes  from  the  more  familiar  form  of  Bayes’  Theorem,  from 


P(A  |  A)  = 


'EPiAjWAlAj) 


(A3) 


j= i 

It  should  be  recalled  that  the  TPT  applies  to  mutually  exclusive  events.  The  TPT  states 
that  the  denominator  of  equation  (A3)  is  equal  to  P(A).  The  mutually  exclusive  events 
are  the  A/s. 


APPENDIX  B  -  MEASUREMENT  TIME  UPDATE  FROM  BAYES  RULE  AND 
THE  TOTAL  PROBABILITY  THEOREM 


Here  is  the  derivation  of  equation  (2),  starting  from  Bayes’  rule. 


P(A  |  B )  =  P{AB^ 

P(B) 

(Bl) 

P(B\A)=P M 

P(A) 

(B2) 

Solving  for  P(AB)  in  equations  (Bl)  and  substituting  the  result  into  equation  (B2)  yields 

the  following  equation. 

P(A) 

(B3) 

The  total  probability  theorem  states  the  following. 

P(A)  =  fjP(Bi)P(A\Bi) 

i=i 

(B4) 
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Substituting  this  into  equation  (B3)  yields  the  following  equation. 

P(B)P(A  |  B) 


P(B\A)  = 


i= 1 


Let  B=x  and  A=z. 


P(x  I  z)  =  ■ 


P(x)P(z  |  x) 
^P(x,.)P(z|x,.) 


i=l 


Rewrite  this  as  follows. 


P(x  I  z)  =  ■ 


P(z  |  x)P(x) 
^P(x,.)P(z|x,.) 


i=l 


Letting  the  summation  become  an  integral. 


P(x  I  z)  = 


Let  x=Xyt  and  z=z^ . 


lz*)  = 


P(z  |  x)P(x) 

J  P{x)P{z  |  x)dx 

^(zi-  I**  )**(**) 


J^t)^(z*  \xk)dxk 


Let  P(xk)=  P(xk\  z1:k-i). 


p(xk  I  zk)  = 


P(Zk  I  *k)P(xk  I  %-_!) 

J^(**  lzi:*-i)^(z*  I  xk)dxk 


Rewriting  this  as  follows. 


p(xk  I  zk)  = 


P(zk  I  xk)P(xk  I  zu-1) 

J  ^(z*  |xt)P(x* 


Let  P(xk\zk)=  P(xk\  z1:k). 


P(xk  I  zl*)  = 


P(zk  \xk)P(xk  I  ZgJ 
Jr(za,  |xa)P(xa  z1:t_, ) Jxa 


(B5) 


(B6) 


(B7) 


(B8) 


(B9) 


(BIO) 


(Bll) 


(B12) 


This  is  in  now  the  form  of  equation  (2). 

APPENDIX  C  -  CHAPMAN  KOLMOGOROV  EQUATION 

Here  we  are  basically  using  a  transitive  property  of  integrating  probability  density 
functions. 
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p(xA  |JCC)  =  J p(xA  I  xB)p{xB  I  xc)dxB  (Cl) 

This  easily  translates  to  equation  (1).  Markov  chains  are  sequences  of  random  variables 
in  which  the  future  variable  is  determined  by  the  present  variable  but  is  independent  of 
the  way  in  which  the  present  state  arose  from  its  predecessors. 

APPENDIX  D  -  WEIGHTED  SUM  OF  DELTA  FUNCTIONS 

The  main  idea  of  a  particle  filter  is  to  approximate  p(xk\zj:j)  with  a  sum  of  Dirac 
Delta  functions  located  at  the  samples  xfi). 

N 

p(xk  |  zvk  )  =  Yj  d,S(xk  -  xk  (/))  (D 1) 

1=1 

Where  the  qt  weights  are  valid  at  the  kth  time  step  and  are  given  by  equation  (3). 

APPENDIX  E  -  IDEAS  FOR  NETWORKED  SENSOR  MANAGEMENT 

Ideas  Presented  for  Networked  Sensor  Management 
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Abstract  The  theory  of  data  fusion  has  become  important  to  military  applications  and  in 
particular,  tracking  applications.  Discussed  are  possible  tracking  processes  for  each  of  the 
first  three  levels  of  data  fusion.  An  introduction  to  data  fusion  and  the  different  levels  are 
included. 

Introduction  to  Information  Fusion 

‘’’Data  Fusion  is  a  process  dealing  with  the  association,  correlation,  and 
combination  of  data  an  information  from  single  and  multiple  sources  to  achieve  reined 
position  and  identity  estimates  for  observed  entities,  and  to  achieve  complete  and  timely 
assessments  of  situations,  threats  and  their  significance. 
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The  process  is  characterized  by  continuous  refinements  of  its  estimates  and 
assessments,  and  by  the  evaluation  of  the  need  for  additional  sources,  or  modification  of 
the  process  itself,  to  achieve  improved  results.”  JDL  DFG  1992’  (10) 

Level  1  Fusion:  Object  Refinement 

“Observation-to-track  association,  continuous  state  estimation  (e.g.  kinematics), 
discrete  state  estimate  (e.g.  target  type  and  ID)  and  prediction.”  (10) 

Level  2  Fusion:  Situation  Refinement 

“Object  clustering  and  relational  analysis,  to  include  force  structure  and  cross 
force  relations,  communications,  physical  context,  etc.”  (10) 

Level  3  Fusion:  Impact  Assessment 

“[Threat  refinement]:  threat  intent  estimation,  [event  prediction],  consequence 
prediction,  susceptibility,  and  vulnerability  assessment.”  (10) 

Levels  0,  4,  and  5  Fusion: 

Level  0:  Sub-object  data  association  and  estimation. 

Level  4:  Process  refinement  (resource  management). 

Level  5:  Human/computer  processing. 

Discussion  of  Ideas  for  Levels  1  through  3  Information  Fusion 
Level  1  Object  Refinement 

The  Kalman  Filter  has  been  widely  used  for  infonnation  fusion 
applications.  This  dynamic  estimation  algorithm  has  had  many  filters  modeled  after  it. 

“The  Kalman  filter  addresses  the  general  problem  of  trying  to  estimate  the  state  '  ~  91  of 
a  discrete-time  controlled  process  that  is  governed  by  the  linear  stochastic  difference 
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equation  + 1  ”  +  +Mj'  with  a  measurement  that  is  l‘;  ~  1^  'Jr  + ' * 

(http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/WELCH/kalman.Lhtml) 


Figure  4.2:  A  complete  picture  of  the  operation  of  the  Kalman  filter,  combining  the 
high-level  diagram  of  Figure  4.1  with  the  equations  from  table  4.1  and  table  4.2. 

http://www.cs.uns.edu/~tracker/media/pdf/SIGGRAPH2001_CoursePack_08.pdf 

MATLAB  for  Kalman  Filtering 

During  summer  of  2004  research,  potential  MATLAB  solutions  were  found  for 
Kalman  Filter  tracking  applications.  A  working  example  of  the  Kalman  Filter  for 
MATLAB  was  found  as  free  software.  The  software  can  be  implemented  to  various 
degrees  for  target  tracking.  It  is  also  possible  to  modify  the  software  for  simple  to 
medium  complex  situations.  Author:  Kevin  Murphy,  1998,  Affiliation:  MIT  Artificial 
Intelligence  Laboratory,  http://www.ai.mit.edu/~murphyk/software/kalman/kalman.html 
In  addition  to  this  software,  a  valuable  resource  was  the  text  “Estimation  with 
Applications  to  Tracking  and  Navigation”  by  Bar  Shalom,  et  al.  Bar  Shalom  and 
colleagues  discuss  what  is  called  the  IMM  or  Interacting  Multiple  Model  estimator.  This 
IMM  is  based  on  the  Kalman  Filter.  “In  the  interacting  multiple  model  (IMM)  estimator, 
at  time  k  the  state  estimate  is  computed  under  each  possible  current  model  using  r  fdters, 


13 


with  each  filter  using  a  different  combination  of  the  previous  model-conditioned 
estimates-  mixed  initial  condition.''’  (11) 

The  use  of  the  IMM  estimator  could  be  applied  easily  to  ATC  (air  traffic  control) 
tracking  and  estimation.  Bar  Shalom  discusses  this  application  in  two  modes  of  flight- 
uniform  motion  and  maneuver.  Results  received  from  a  sample  problem  in  the  text  were 
as  follows: 

“The  KF  is  clearly  inferior  to  any  (reasonably  designed)  IMM  estimator.  The 
accuracy  of  the  turn  rate  estimate  is  not  very  important  as  far  as  the  quality  of  the 
position,  speed,  and  course  estimates  are  concerned.  What  is  important  is  the  correct  and 
timely  detection  of  the  maneuver  and  the  fast  response  of  the  filter  to  his  direction.  The 
IMM-CT  estimator  is  the  best  choice  for  tracking  maneuvering  with  its  capability  to  track 
the  linear  as  well  as  turn  motion  of  the  target.”  (11) 

Future  exploration  on  the  topic  of  level  1  fusion  would  include  using  the 
combined  knowledge  of  Kalman  filtering  and  IMM  estimators.  Using  this  knowledge, 
one  can  develop  software  using  Kevin  Murphy’s  Kalman  Toolbox  and  Bar  Shalom’s,  et 
al  DynaEst  (a  MATLAB  program  developed  for  the  IMM)  to  perfonn  accurate  and  fast 
tracking  estimations  for  ATC  applications. 

Level  2  Situation  Refinement 

The  approach  for  level  2  fusion  was  researching  multi-image  fusion.  The  text 
“Multiple  View  geometry  in  Computer  Vision”  by  R.  Hartley  and  A.  Zissennan  was  the 
primary  source  of  research.  Research  was  concentrated  on  Trifocal  Tensor  Estimation 
and  Bundle  Adjustment.  The  goal  was  to  find  an  appropriate  model  to  develop  3D 
reconstructions  from  2D  images  and  identify  targets  using  the  retrieved  images. 
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Trifocal  tensor  estimation  uses  multiple  2D  images  and  tensor  algebra  to  form  a  3 
dimensional  result. 

Example: 

•  Obtained  3  separate  images  of  the  same  object  at  3  slightly  different  angles. 

•  Fundamental  matrices  may  not  always  work  when  reconstructing  the  images  into 
a  scene. 

•  Need  to  estimate  3  projections  at  once. 

•  Deployment  of  a  trifocal  tensor  can  be  used  for  this  process  where  each  “camera” 
view  is  related  by  a  fundamental  matrix. 

Trifocal  tensors,  along  with  fundamental  matrices,  can  be  very  useful  in  computer  vision 
when  building  3D  images  from  large  sequences  of  images  (20  or  30  or  more).  (12)  The 
following  figure  shows  the  basic  construction  of  the  trifocal  plane. 


Trifocal  tensor  algebra  is  complicated,  but  it  can  be  used  to  fuse  many  images  retrieved 
from  networked  sensor  equipment.  Possible  types  of  networks  this  could  be  used  in  are 
Multi-UAV  with  ground  control,  UAV  to  ground,  Multi- Aircraft,  etc. 
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Bundle  adjustment  is  even  more  complicated,  but  may  be  more  accurate  when 
reconstructing  into  3  dimensions.  “Bundle  adjustment  is  the  problem  of  refining  a  visual 
reconstruction  to  produce  jointly  optimal  3D  structure  and  viewing  parameter  (camera 
pose  and/or  calibration)  estimates.  Optimal  means  that  the  parameter  estimates  are  found 
by  minimizing  some  cost  function  that  quantifies  the  model  fitting  error,  and  jointly  that 
the  solution  is  simultaneously  optimal  with  respect  to  both  structure  and  camera 
variations.”  (13)  This  obviously  would  have  more  complicated  mathematics  that  trifocal 
tensor  estimation.  However,  the  capability  to  account  for  error  could  be  of  more  use  for 
military  applications  that  trifocal  tensor  estimation. 

Level  3  Impact  Assessment 

This  is  one  of  the  most  important,  and  difficult  to  find  solutions  for,  levels  of 
fusion.  It  is  very  important  to  target  tracking  since  it  is  difficult  to  know  what  an  enemy 
vehicle  or  even  friendly  vehicle  may  do.  Past  attempts  have  shown  that  impact 
assessment  can  be  complicated  to  implement  or  to  develop  algorithms.  The  majority  of 
research  for  all  levels  of  fusion  have  been  concentrated  on  impact  assessment  more 
recently  because  of  this. 

The  proposed  solution  researched  over  the  summer  of  2004  and  continued  from 
27  December  2004  to  21  January  2005  is  Dynamic  Game  Theory.  Game  theory  has 
become  extremely  popular  for  evolutionary  AI  purposes.  Most  of  the  purposes  allow  AI 
programs  to  “evolve”  as  they  work.  However,  not  much  research  has  been  performed 
with  respect  to  level  3  fusion.  One  first  has  to  have  a  basic  knowledge  of  game  theory  and 
realize  that  impact  assessment  in  an  air  combat  “game”  is  not  a  cooperative  one  as  well  as 
being  dynamic.  This  basic  knowledge  makes  the  available  text  more  understandable. 
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Summer  Research 


There  has  been  research  performed  in  the  area  of  combat  games  by  few 
researchers.  Pursuit  evasion  games  have  had  text  available  since  the  “acknowledged 
father  of  pursuit  evasion  games,”  R.  Isaacs,  published  “Differential  Games”  in  1965.  In 
pursuit  evasion  games,  the  time  duration  of  the  game  is  not  fixed.  Basar  and  Olsder 
discuss  pursuit  evasion  games  in  their  text,  “Dynamic  Noncooperative  Game  Theory.” 
(14)  A  game  of  this  type  uses  simple  kinematics  but  complicated  game  theory 
mathematics  to  calculate  the  “threat”  of  a  target  by  evaluating  whether  a  vehicle  is  in 
pursuit  or  evasion.  If  this  process  was  combined  with  target  tracking  via  Kalman  Filtering 
techniques  and  multi-view  geometry,  it  would  give  more  accurate  threat  assessments. 

Conclusions  during  the  summer  showed  that  more  basic  game  theory  analysis 
may  be  used  to  assess  threat  using  Kalman  filtering  and  multi-view  geometry  retrieved 
information.  Sensor  management  could  be  carried  out  in  such  a  way  that  certain  sensors 
hold  precedence  over  others  in  certain  situations.  This  would  be  controlled  through  Nash 
Bargaining  theory. 

An  example  of  an  air  combat  game  is  shown  below. 


(15) 
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This  pursuit  evasion  game  involves  red  and  blue  forces.  The  game  model  offers  a  novel 
way  of  analyzing  optimal  air  combat  maneuvering  and  to  develop  automated  decision 
making  system  for  selecting  combat  maneuvers. 

The  following  figure  allows  one  to  see  who  gains  or  loses  the  competitive  advantage  over 
time  in  the  above  game. 

BLUE  RED 


(15) 


Level  3  fusion  can  be  accomplished  by  plotting  the  probability  of  the  threat  situation 
outcome  as  a  function  of  time.  Game  theory  aids  us  in  performing  important  tasks  in  level 
3  fusion. 

Winter  Research 

Pareto  Optimality 

The  idea  of pareto  optimality  is  widespread  in  game  theory.  The  concept  is 
defined  as  a  way  to  make  at  least  one  “player”  of  a  game  better  off  without  making  any 
other  “player”  worse  off  than  they  were.  This  is  an  obvious  plus  for  air  force  or  other 
military  applications.  To  better  understand  what  this  means,  an  example  follows.  Given  a 
group  of  ground  vehicles  and  one  aircraft,  if  the  aircraft  has  the  optimal  field  of  view  of  a 
target,  the  aircraft  controller  could  then  send  information  to  the  ground  vehicles  and 
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therefore  no  one  on  what  can  be  called  the  blue  team  will  be  at  more  of  a  disadvantage 
than  they  would  have  if  the  target  (or  red  team)  information  was  not  known.  This  is  a 
very  basic  example,  but  it  is  possible  to  broaden  the  idea  to  complicated  situations  such 
as  where  multiple  UAV’s  are  present  and  information  is  needed  on  whether  or  not  one 
particular  UAV  is  transmitting  more  important  information  than  another. 

In  the  presence  of  a  combat  “game,”  impact  assessment  must  be  maintained  in 
order  to  decide  what  is  optimal.  By  combining  a  pursuit  evasion  game  to  predict  positions 
of  targets  and  a  pareto  optimality  strategy,  one  can  assess  the  threat  of  the  target  in 
addition  to  which  “blue”  vehicle  should  hold  precedence  when  tracking  the  target. 

Group  Target  Tracking 

Sadjadi  and  Kober  (16)  discuss  group  target  tracking  using  game  theory  in  terms 
of  state  vectors  and  game  theoretic  leader-follower  strategies.  Assumptions  are  made  that 
object  are  members  of  one  single  group  and  their  state  vectors  are  affected  by  terrain 
topography  and  group-membership  requirements.  To  determine  a  group,  a  matrix  can  be 
set  up  using  x,  y,  and  z  (when  needed)  axes  with  the  velocity  vector  projection  associated 
with  a  different  target  in  each  row.  By  finding  the  covariance  associated  with  the  velocity 
matrix  and  finding  the  eigenvalues,  one  can  see  whether  or  not  the  targets  are  moving  in 
the  same  direction  and  consequently,  if  they  belong  to  the  same  group.  Likewise,  the 
eigenvectors  of  the  covariance  matrix  can  show  the  general  direction  that  the  group  is 
moving. 

It  is  discussed  that  if  the  eigenvalues  are  both  distinct,  that  2  situations  may  be 
occurring.  One  is  that  the  group  is  splitting  into  two  groups  and  moving  separately.  The 
second  is  that  the  group  remains  as  one  but  the  terrain  is  causing  a  sequential  path  change 
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of  parts  of  the  group  to  be  curved.  The  leader-follower  model  later  discussed  deals  with 
combining  the  eigenvalue  analysis  with  the  model. 

The  leader-follower  model  is  exactly  how  it  sounds.  There  will  almost  always  be 
a  “leader”  of  a  group  and  each  individual  object  will  follow  that  leader.  Because  of  this, 
game  theoretical  computations  can  be  made  directed  towards  the  state  of  the  group.  This 
all  comes  back  to  the  IMM  estimator  discussed  previously.  Worth  mentioning  is  the  fact 
that  in  order  to  properly  use  this  group  target  tracking  technique,  there  must  be  certain 
control  laws  assigned  to  group  members  that  are  used  to  preserve  group  membership.  In 
other  words,  there  must  be  a  leader  and  there  must  be  a  follower. 

Grocholsky  Thesis 

Ben  Grocholsky  of  the  University  of  Sydney  has  written  a  thesis  based  on  multi¬ 
sensor,  multi-vehicle  systems.  The  thesis  discusses  “principles  and  architecture 
developed  for  decentralized  data  fusion  and  sensor  management.”  (19)  Grocholsky 
mainly  discusses  the  autonomous  operation  of  robotic  systems  and  how  they  collectively 
work  together  towards  a  common  goal  via  sensor  management.  However,  the  same  logic 
can  be  applied  to  human  operation.  The  best  way  to  explain  the  thesis  is  to  simply  quote 
the  author. 

•  Chapter  2  considers  approaches  to  distributed  multi-robot  systems.  Conventions 
used  through  the  thesis  are  stated.  These  include  definitions  of  coordination, 
cooperation  and  the  characteristics  of  decentralized  systems.  The  formulation  of 
the  team  decision  problem  is  presented  and  its  connection  to  the  Nash  bargaining 
problem  is  established.  An  iterative  procedure  known  as  better-response 
negotiation  is  introduced  as  a  means  for  determining  Nash  equilibria.  Key 
elements  in  engineering  decentralized  decision  making  team  members  are 
identified  as:  modeling  of  the  environment,  sensors  and  vehicles;  specification  of 
communication  structures;  capturing  team  utility;  parameterization  of  actions  and 
devising  solution  procedures. 

•  Chapter  3  covers  the  problem  of  quantifying  and  fusing  infonnation  in  multi¬ 
sensor  systems.  Information  is  formally  defined,  in  tenns  of  uncertainty,  by  Fisher 
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and  Shannon  measures.  The  Information  filter  is  presented  as  a  mechanism  for 
scalable  decentralized  fusion  of  data  from  multiple  sources.  The  manner  in  which 
information  is  lost  and  gained  in  the  fusion  process  is  discussed  and  quantified. 
Entropic  and  mutual  information  are  determined  to  be  appropriate  expected  utility 
measures  for  sensing  actions.  Common  information  among  observations  is 
identified  as  the  source  of  coupling  in  team  utility  derived  from  entropic 
information.  The  decentralized  data  fusion  process  and  infonnation-theoretic 
utility  structure  are  identified  as  fonning  a  consistent  basis  for  gathering,  ex¬ 
changing,  evaluating  and  fusing  information  in  the  team  decision  problem.  The 
approach  is  demonstrated  through  the  analysis  of  a  discrete  sensor  assignment 
problem. 

•  Chapter  4  presents  information  gathering  as  an  optimal  control  problem. 

Modeling  of  the  environment,  vehicles  and  sensors  is  combined  with  utility  based 
on  entropic  information.  This  is  applied  to  the  determination  of  optimal 
information  seeking  trajectories  for  7  the  case  of  a  single  bearings-only  sensor 
platform  localizing  a  point  feature.  The  implications  of  this  example  for  active 
sensing  tasks  is  explored  and  discussed.  Consideration  then  turns  to  problems 
involving  multiple  sensor  platforms.  Attention  is  focused  on  the  team  utility 
structure  and  its  role  in  cooperation.  A  proposed  decomposition  of  the  team  utility 
is  used  to  explore  the  influence  of  coupled  utility  on  the  optimal  member 
decisions.  This  identifies  relationships  between  the  optimal  individual  and  team 
solutions  with  implications  for  the  complexity  of  the  cooperative  solution.  A 
localization  example  with  two  range-only  sensors  is  used  to  illustrate  these 
results.  It  is  then  demonstrated  that  the  optimal  team  solution  can  be  determined 
through  a  better-response  negotiation  procedure. 

•  Chapter  5  explores  communication  and  coupled  utility  among  decision  makers  as 
fundamental  mechanisms  underlying  coordination  and  cooperation.  Propagation 
of  observation  information  through  the  decentralized  data  fusion  process  leads  to 
coordination  by  altering  the  prior  information  on  which  local  decisions  are  based. 
The  individual  decision  making  processes  become  coupled  when  propagation  of 
expected  observation  information  is  permitted.  This  enables  determination  of  the 
cooperative  team  solution  by  negotiation.  Coordinated  and  cooperative  solutions 
are  demonstrated  through  extension  of  the  single  vehicle  bearings-only  example 
from  Chapter  4  to  multiple  sensor  platforms  and  features.  The  applicability  of  this 
approach  to  other  tasks  is  demonstrated  through  an  area  exploration  problem. 
Finally,  all  the  elements  considered  through  this  are  brought  together  to  fonn  a 
general  architecture  for  decentralized  coordinated  control  of  multi-sensor 
information  gathering  systems. 

•  Chapter  6  presents  the  main  conclusions  and  identifies  a  range  of  future  research 
directions  for  the  work  described  in  this  thesis.  (19) 

As  one  can  see,  this  is  a  direct  approach  to  the  difficulty  of  data  fusion.  Level  3 
fusion  is  once  again  discussed  in  a  manner  of  game  theory  using  the  Nash  bargaining 
problem  approach.  The  optimal  control  problem  discussed  is  representative  of  level  3 
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fusion,  also.  This  refers  to  the  idea  that  one  sensor  may  hold  precedence  over  all  others 
due  to  information  being  gathered  showing  more  importance.  Overall,  the  thesis 
discusses  novel  ways  to  approach  data  fusion  using  optimal  control  theory  and 
information-theoretic  techniques.  These  are  highly  relevant  techniques,  especially  for 
multi-UAV  systems. 

Particle  Filtering  (also  discussed  by  Mark  Alford) 

The  method  of  particle  filtering  is  a  Bayesian  state  estimate  based  system  for 
nonlinear/non-Gaussian  problems.  One  of  the  most  popular  approaches  to  the  problem  is 
using  the  EKF  (extended  Kalman  filter).  However,  this  approach  still  requires  a  Gaussian 
PDF  (probability  density  function)  at  every  iteration.  This  is  an  issue  because  the 
nonlinear/non-Gaussian  PDF  is  not  a  general  closed  form  expression.  Gordon,  et  al  (17) 
discuss  other  methods  that  have  been  attempted.  Presented  is  a  filter  labeled  the 
“bootstrap”  filter  with  implement  a  recursive  Bayesian  filter. 

The  main  idea  is  to  represent  the  PDF  as  a  set  of  random  variables  rather  than  as  a 
function.  Since  a  large  number  of  samples  can  be  obtained,  accuracy  can  be  optimal. 
Without  becoming  overindulged  in  the  mathematics,  it  should  be  said  that  particle 
filtering  methods  prove  to  be  accurate  against  other  filters.  (18)  Several  types  of  particles 
filters  have  been  developed,  including  a  grid-based  filter  which  is  based  on  a  numerical 
integration  method,  all  seem  to  show  improved  outcome  over  the  next  best,  EKF. 

If  these  techniques  are  applied  to  game  theory  for  level  3  fusion,  one  may  find 
that  the  accuracy  of  impact  assessment  will  increase.  The  method  can  be  compared  to  the 
group  target  tracking  discussion.  If  the  same  system  is  applied  here  but  using  a  particle 
filter,  it  may  be  seen  that  the  accuracy  will  increase  tremendously  over  using  a  simple 
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Kalman  filter  technique.  This  is  so  because  most  real  life  situations  in  target  tracking  are 
nonlinear  and  non-Gaussian  which  particle  filters  take  into  account. 

Mahler’s  Force  Aggregation 

Ronald  Mahler  of  Lockheed  Martin  Tactical  Systems  has  developed  a  unified 
approach  to  force  aggregation,  or  level  2  data  fusion.(20,21)  This  approach  is  similar  to 
the  particle  filtering  technique  described  above  using  a  recursive  Bayesian  filter.  Mahler 
applies  the  filter  to  a  multi-sensor,  multi-target  group.  However,  instead  of  the  PDF  being 
used,  Mahler  uses  the  probability  hypothesis  density  (PHD).  The  aforementioned  particle 
filtering  only  deals  with  continuous  target  groups,  however,  and  not  with  situations  where 
parts  of  the  group  may  separate. 

Mahler  discusses  equations  involved  with  his  recursive  filter  and  compares  it  to 
others.  He  introduces  the  random  cluster  process  with  reference  to  group  tracks.  This  is 
so  because  there  are  in  affect,  three  layers  to  the  group  tracking  process-  a  “twice-hidden 
group  target  layer,”  “a  single-hidden  layer,”  and  a  “visible  observation  layer.”  There  is  an 
underlying  “mother  process”  which  consists  of  finite  definitions.  In  simple  terms,  the 
total  statistical  representation  of  a  multi-sensor,  multi-group  system  can  be  defined  as  a 
random  finite  subset. 

It  is  shown  that  Mahler’s  process  is  accurate  for  state  estimation  with  regards  to 
level  2  data  fusion.  However,  the  assumption  should  be  noted  about  the  need  that  there 
should  be  an  approximation  strategy  due  to  the  generalization  of  the  PHD.  This  is 
because  the  multi-sensor,  multi-group  Bayes  filtering  equations  defined  by  Mahler  are 
computationally  expensive. 
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Closing  Statement 

Much  information  was  collected  that  could  potentially  create  ease  of  data  fusion 
in  the  future.  There  are  many  options  open  for  level  3  fusion.  These  ideas  should  be 
investigated  further  so  as  to  build  a  strong  foundation  for  impact  assessment.  Together 
with  the  research  found  building  levels  one  and  two  fusion,  impact  assessment  can  be 
accurately  detected. 
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APPENDIX  F  -  WORK  PLAN  FOR  IN-HOUSE  RESEARCH 


WORK  PLAN 
FOR 

IN-HOUSE  RESEARCH 

DEVELOPMENT  AND  EVALUATION  OF 
FUSION  TECHNIQUES  (DEFT) 

Mark  Alford  x3802  Eric  Jones  x4410  Adnan  Bubalo  x2991 


Introduction 


The  Development  and  Evaluation  of  Fusion  Techniques  (DEFT)  in-house  program  has 
multiple  purposes.  These  purposes  will  be  elaborated  in  Part  I  and  Part  II  below.  This 
effort  is  providing  manpower  only  to  establish  an  in-house  program  in  fusion.  A  brief 
introduction  to  the  DEFT  program  is  as  follows: 

The  first  purpose  of  DEFT  is  to  extend  the  development  of  algorithmic  concepts  that 
were  identified  and  explored  under  the  Complementary  Advanced  Fusion  Exploration 
(CAFE)  in-house  program.  CAFE  investigated  promising  recent  innovations,  primarily 
involving  mathematical  concepts  within  the  area  of  basic  research,  and  did  a  preliminary 
assessment  of  their  validity  as  well  as  a  preliminary  qualitative  assessment  of  each 
concept’s  potential  for  application  to  Air  Force  Command,  Control,  Intelligence, 
Reconnaissance,  and  Surveillance  (C2ISR)  capabilities  and  systems.  DEFT  will  also 
continue  the  investigation  of  potential  exploratory  development  applications  of  the  CAFE 
algorithmic  concepts.  In  addition,  DEFT  will  also  investigate  concepts  that  are  revealed 
as  extensions  of  the  original  CAFE  concepts. 

The  second  purpose  of  DEFT  is  to  investigate  additional  concepts  that  were  not  originally 
investigated  under  CAFE  for  their  exploratory  development  potential.  Many  concepts 
have  been  developed,  particularly  through  the  Small  Business  Innovation  Research 
(SBIR)  program.  Often,  these  concepts  are  developed  to  show  basic  feasibility  and 
validity  of  the  concepts  without  fully  investigating  their  exploratory  development 
potential  for  application  to  Air  Force  C2ISR  capabilities  and  systems.  DEFT  will 
examine  three  such  areas  which  appear  to  have  this  exploratory  development  potential. 
In  addition,  DEFT  will  examine  other  concept  areas  of  this  type  as  they  are  identified. 

DEFT  will  be  accomplished  is  a  systematic  manner.  This  will  include  extensive  use  of 
existing  publications.  Concepts  will  be  further  investigated  through  mathematical 
verification  as  required,  mathematical  development  as  required,  and  incorporation  into 


25 


and  exercising  of  computer  simulations.  Limited  creation  of  code  will  be  pursued  when 
necessary  to  support  the  simulation  process.  Documentation  will  be  accomplished 
through  periodic  technical  memos  and  reports  as  well  as  through  the  development  of 
papers  for  journals  and/or  symposia  as  opportunities  present  themselves  and  when  results 
justify  it.  Concepts  that  are  fully  developed,  verified,  and  that  show  potential  for  useful 
Air  Force  C2ISR  exploitation  will  be  further  developed  and  presented  to  potential 
funding  sources  for  further  formal  exploratory  development  efforts  and  other  types  of 
technology  transition. 


Part  I:  CAFE  Extensions 


As  stated  in  the  introduction,  DEFT  will  extend  the  development  of  algorithmic  concepts 
that  were  identified  and  explored  under  CAFE.  CAFE  investigated  a  number  of 
promising  recent  innovations,  a  number  of  which  are  described  below. 

This  part  of  DEFT  will  complement  existing  and  new  fusion  research  arenas.  It  will 
explore  new  technologies  to  complement  work  done  in  past  and  on-going  contracts.  A 
framework  for  past,  existing  and  future  efforts  will  be  provided. 

In  addition,  this  part  of  DEFT  will  involve  exploration  of  new  mathematical  and  physical 
areas,  concepts  and  ideas  to  apply  to  fusion  technology.  A  major  focus  will  be  to  review 
the  most  recently  published  papers  to  determine  which  ones  have  the  most  application  to 
information  fusion.  Higher  level  mathematical  constructs  will  be  documented  for  later 
application  to  specific  fusion  challenges.  New  fusion  concepts  will  be  investigated. 
Mathematical  models  will  be  examined  and  refined  for  application  to  real  world 
problems. 

Research  focus  will  initially  be  directed  towards  a  number  of  the  concepts  explored  under 
CAFE.  These  include: 


-  Tensors  (including  Homography) 


Trilinear  tensors  are  used  for 
reprojection  of  images 


3 

Reprojection  is  taking  an  image  and  reprojecting 
(aka  warping,  remapping)  it  to  a  more  useful  map 
coordinate  system. 


Tensors:  Potential  Payoff 


http://www.cognitens.  com/pages/Products. asp?intGloballD=48 
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Nash  (Grocholsky)  Approach 

Nash  Equilibrium  Approach  to  Sensor  Management 

What  allows  a  collective  of  distributed  autonomous  decision  makers  to  work  together 
toward  a  common  objective?  The  problem  of  seeking,  sensing,  interpreting  perceptual 
information  and  interacting  with  other  decision  making  elements  in  an  inherently 
uncertain  environment  is  a  complex,  and,  as  yet,  unsolved  problem. 

A  team  consists  of  multiple  decision  makers.  Each  decision  maker  must  make  a  decision 
that  accounts  for  the  decisions  made  by  other  members  of  the  team.  The  team  decision 
problem  is  to  find  optimal  decision  rules  for  each  member  so  that  the  expected  utility 
(payoff)  of  the  whole  team  is  maximized. 

This  cooperative  situation  falls  under  the  general  category  of  bargaining  problems 
defined  by  Nash.  It  is  a  situation  where  individuals  take  actions  associated  with  a  set  of 
outcomes.  Each  individual  desires  to  maximize  their  gain  from  a  bargaining  process. 
The  individuals  are  able  to  accurately  compare  preferences.  The  exchange  is  cooperative 
in  the  sense  that  individuals  are  able  to  discuss  and  agree  on  a  joint  plan  of  action. 

Nash  Approach  Payoff 


From  Grocholsky’s  Thesis 


Considerations: 

What  do  we  do  when  the  platforms  do  not  exchange  information? 

How  do  we  handle  situations  when  there  are  conflicts  between  the  sensors? 
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-  Virtanen  Approach 


JSF  pursuit  evasion  game 


Air  combat  modeling 


Applies  to  other  scenarios  as  well 


Pursuit-evasion  situation  involving  red  and  blue  forces 

This  model  offers  a  novel  way  to  analyze  optimal  air  combat 
maneuvering  and  to  develop  an  automated  decision  making 
system  for  selecting  combat  Maneuvers. 


Virtanen  Methodology  Payoff 


Gaining  a  competitive  advantage:  Level  III  fusion  can  be 
accomplished  by  plotting  probability  of  threat  situation 
Outcome  as  a  function  of  time. 


-  OOSMs,  OOSTs 

Out  of  Sequence  Problem  (OOSP) 

Most  Information  Fusion  Filters  assume  sequential  information  arriving  in  a  particular 
order.  In  many  cases,  in  the  real  world,  data  (measurements,  tracks,  etc.)  arrives  out  of 
order  or  out  of  sequence.  Several  papers  address  this  problem,  one  being  the  thesis  by 
Keshu  Zhang.  In  her  thesis,  Keshu  uses  Best  Linear  Unbiased  Estimation  (BLUE) 
Fusion.  She  addresses  the  problem  of  how  to  update  state  estimates  with  Out  of 
Sequence  Measurements  (OOSM).  She  also  addresses  how  to  apply  the  OOSM  update 
algorithm  to  multiple  targets  in  clutter 
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00  SM  Updating 

Suppose  there  is  no  storage  constraint.  Then  the  OOSM  update  problem  can  be  simply 
solved  by  rerun  the  Kalman  filter  with  all  data  from  the  OOSM  occurrence  time  to  its 
arrival  time.  Bar  Shalom  first  derived  the  OOSM  update  algorithm  for  the  one-step  delay 
problem. 


OOSM  Payoff,  2  GMTI  Radars 


-  Particle  Filters  (PF) 

Overcomes  Gaussian  and  Linear  assumptions  of  the  Kalman  Filter.  It  is  based  on  the 
Chapman  Kolmogorov  Equation  (prediction)  and  Bayes  Theorem  (update).  PF  models 
the  target  states  as  a  sum  of  weighted  delta  functions  (particles).  Particle  weights  are 
computed  from  samples  of  a  Monte  Carlo  distribution. 

New  variants  of  the  basic  bootstrap  PF  are  being  developed  almost  daily.  Examples 
include:  Sequential  Importance  Sampling,  Re-Sampling,  Generic  PF,  Sequential 
Importance  Re-sampling  PF,  Auxiliary  PF,  Regularized  PF,  Likelihood  PF,  and  PHD  PF. 

These  are  but  a  few  of  the  growing  number  of  approaches.  Goal  of  current  research  is  to 
review  the  various  algorithms  and  get  a  basic  understanding  of  the  PF  process.  Efforts 
are  underway  by  researchers  to  combine  Identification  and  PF  track  fusion  and  also  to 
develop  a  basis  for  assigning  particular  PF  algorithms  to  particular  types  of  real  world 
problems. 
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Ground  Target  Tracking  Using  an 
Time  and  Measurement  Update  Airborne  Particle  Filter 


J.  Schubert,  C.  Martenson,  H.  Sidenbladh,  P.  Svenson,  J.  Walter,  “Methods 
and  System  Design  of  the  FOI  Information  Fusion  Demonstrator - 
IFD03,”  ICCRTS  9th  International  Command  and  Control  Research  and 
Technology  Symposium  September  14-16,  2004,  Copenhagen,  Denmark. 


Payoff  of  PF  Compared  to  EKF 

From  tutorial  by  Arulampalam 


Algorithm 

Proposal 

Section 

Figures 

RMSE 

Extended  Kalman  filter 

N/A 

IV-A 

3.4 

23.19 

Approximate  grid-Based  Filter 

N/A 

IV-B 

5 

6.09 

SIR  Particle  filter 

V-B.l 

6 

5.54 

Auxiliary  Particle  filter 

V-B.2 

7 

5.35 

Regularised  Particle  filter 

p(x*K_,) 

V-B.3 

8 

5.55 

‘Likelihood’  Particle  filter 

p(3Cfc|Sfc)p(sfc|zjt) 

A 

9 

5.30 

Likelihood  PF  has  lowest  Root  Mean  Squared  Error  5.30  as  compared 

to  23.19 


-  Other  Topics 

More  topics  will  be  discovered  and  pursued  during  this  effort. 

Some  new  ideas  that  could  be  pursued  in  this  in-house  effort  would  be: 

-  PF  Track  error  -  variation  from  classic  ellipsoidal  error 

-  General  error  &  volume  track  shape  error  measurement 

-  Particle  clustering 

-  S/W  to  test  PFs  for  air-space-ground-sea 

-  Unclassified  PF  testbed  with  high  computational  speeds/storage  capacity 
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Part  II;  Other  Exploratory  Development  Topics 


As  stated  in  the  introduction,  DEFT  will  also  investigate  additional  concepts  that  were 
not  originally  investigated  under  CAFE  for  their  exploratory  development  potential. 
Three  concepts  will  be  investigated;  however  this  will  not  preclude  the  investigation  of 
other  new  concepts  that  are  identified.  The  first  is  an  examination  of  the  potential  for 
advances  in  the  exploitation  of  Bayesian  Networks.  The  second  is  an  examination  of 
ways  to  exploit  techniques  that  allow  for  a  faster  determination  of  optimized  routes  for 
Intelligence,  Surveillance,  and  Reconnaissance  (1SR)  platfonns.  And  the  third  is  an 
examination  of  the  possibilities  for  improved  1SR  capabilities  through  exploitation  of 
Radio  Frequency  (RF)  Tag  technology,  Unattended  Ground  Sensor  (UGS)  technology, 
and  other  associated  techniques. 

This  part  of  DEFT  shall  integrate  the  use  of  analysis  of  concepts  that  have  been 
developed  through  the  study  of  past  results,  extension  of  more  general  analytic  results  for 
use  in  applications  of  interest,  and  simulation  of  concepts  to  investigate  the  utility  of 
legacy  and  developed  concepts.  Investigation  of  the  potential  for  these  concepts  is 
anticipated  to  yield  reasonably  specified  variations  that  can  be  identified  for  further 
potential  exploitation  through  more  formal  Exploratory  Development  activities.  A 
further  description  of  the  three  principal  concepts  under  this  part  of  DEFT  is  as  follows: 

-  Bayesian  Network  Exploitation  for  Fusion 

Bayesian  Networks  have  been  in  use  for  a  number  of  years  now.  Perhaps  the  largest  area 
of  their  use  had  been  in  the  area  of  target  identification  and  discrimination.  More 
recently,  they  have  been  used  more  directly  in  support  of  multi-sensor  fusion.  These 
recent  applications  have  been  showing  increasing  innovation  in  the  expansion  of  the  ways 
in  which  Bayesian  Networks  can  be  used  in  support  of  fusion. 

Among  the  areas  that  have  been  using  Bayesian  Networks  in  these  ways,  the  Missile 
Defense  Agency  (MDA)  is  using  Bayesian  Networks  as  the  basis  for  their  Decision 
Architecture  within  Project  Hercules.  In  addition,  two  recent  SB1R  efforts  have  used 
them  as  a  means  for  multi-sensor  fusion,  in  one  case  to  fuse  data  that  includes  features 
derived  from  the  concept  of  invariants,  and  in  another  as  a  basis  for  accomplishing 
distributed  fusion.  Another  SBIR  effort  used  a  Bayesian  Network  in  association  with  a 
Multiple  Hypothesis  Tracker  (MHT)  as  a  basis  for  target  classification  after  target 
detection  and  tracking  had  been  accomplished  through  the  MHT.  Through  careful 
examination  of  the  various  ways  that  Bayesian  Networks  have  been  used  in  these  and 
other  efforts  and  studies,  it  is  anticipated  that  new  applications  for  them  can  be 
determined. 
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-  Optimization  of  ISR  Platform  Route  Determination 

Trying  to  optimize  the  sensor  tasking  process  for  the  improvement  of  ISR  performance 
for  missions  of  interest  is  an  area  that  has  attracted  much  research  interest  in  recent  years. 

One  approach  has  been  to  try  to  optimize  coverage  through  detailed  up  front  planning, 
usually  in  coordination  with  the  Air  Tasking  Order  (ATO)  process.  This  process  is 
somewhat  slow  but  allows  for  detailed  planning  that  can  utilize  a  large  amount  of 
optimization.  However,  changing  these  plans  based  on  short  term  contingencies  is 
difficult  without  doing  great  damage  to  the  overall  plan. 

On  the  other  hand,  you  can  have  a  much  shorter  planning  window.  This,  however,  while 
it  makes  it  easier  to  allow  for  short-term  contingencies,  results  in  overall  plans  that  are 
considerably  less  optimal. 

Recent  efforts  have  approached  this  problem  from  both  directions.  The  Advanced  ISR 
Management  (AIM)  program  is  a  Defense  Advanced  Research  Projects  Agency 
(DARPA)  program  that  approaches  the  problem  through  the  idea  that  the  coming 
generation  of  collection  systems  will  provide  dramatically  increased  volumes  of  higher 
fidelity  data  to  the  operational  decision-maker.  AIM  then  proceeds  under  the  idea  that 
the  challenge  will  be  to  dynamically  manage  and  synchronize  the  advanced  collection 
architecture  with  next-generation  processing,  exploitation  and  dissemination  capabilities 
to  provide  the  critical  information  to  the  decision  maker  in  the  constantly  changing 
operational  situation.  The  time  to  accomplish  this,  however,  is  to  long,  for  a  scenario  of 
realistic  size,  to  allow  recalculation  of  the  results  based  on  short-term  contingencies. 

A  recent  SBIR  effort,  however,  ISR  Strategy  Optimizer,  demonstrated  the  feasibility  of 
using  mathematical  programming  technology  to  optimize  collection  strategy  selection  for 
scenarios  requiring  multi-platform  coordination  and  synchronization  by  developing 
algorithms  that  were  able  to  quickly  perform  (albeit  abstractly  or  coarsely)  multiple  asset 
path  planning  and  asset-target  assignment  and  scheduling  while  also  selecting  asset  types, 
numbers,  and  bed-down  locations  in  a  global  manner.  Since  ATOs  are  so  interdependent, 
the  ability  to  quickly  recalculate  entire  theater  ISR  platform  assignments  based  on 
unanticipated  contingencies  allows  for  the  possibility  of  much  more  optimal  use  of  ISR 
platform  assets. 

And  a  major  part  of  the  just  completed  DARPA  Dynamic  Tactical  Targeting  (DTT) 
program  as  well  as  the  new  Dynamic  Tactical  Targeting:  Tactical  Exercises  and  System 
Testing  (DTT:TEST)  program  that  is  just  beginning  is  a  portion  of  the  architecture 
devoted  to  Proactive  Sensor  Tasking,  which  will  provide  DTT  with  a  Control  component 
to  continuously  update  plans  for  future  sensor  operations,  including  both  the  routes  that 
platforms  follow,  and  the  collection  tasks  that  sensors  perform. 

It  is  anticipated  that  examination  of  these  efforts  as  well  as  other  efforts  and  studies  will 
help  to  identify  new  forms  of  application  for  ISR  platform  optimal  route  determination. 
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-  RF  Tag  Utilization 


RF  Tag  technology  has  been  under  development  for  many  years.  Great  advances  in  RF 
Tag  technology  were  made  through  the  recent  DARPA  Digital  RF  Tag  (DRaFT) 
program.  Areas  where  RF  Tags  have  been  considered  for  use  include  Blue  Force 
communications  and  tracking.  Particular  consideration  for  the  potential  use  of  RF  Tags 
has  been  to  help  reduce  fratricide.  The  Army’s  current  RF  Tags  Advanced  Concept 
Technology  Demonstration  (ACTD)  is  attempting  to  demonstrate  the  utility  of  the 
DRaFT  device  as  well  as  three  other  RF  Tags  for  these  missions.  DRaFT  was  developed 
with  the  intent  that  it  possesses  great  flexibility,  giving  it  the  ability  to  be  used  for  many 
potential  missions.  Possible  synergistic  devices  and  areas  include  use  with  multiple  ISR 
platforms  working  in  concert  with  RF  Tags,  their  use  in  concert  with  UGS  as  well  as  with 
other  associated  techniques.  Investigation  of  this  area  is  anticipated  to  lead  to  the 
development  of  useful  concepts  for  improved  Blue  Force  communications,  tracking,  and 
other  missions. 
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